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Control Evaluation Structure (MCES) to the Identification Friend, Foe, or Neutral 
(IFFN) Joint Testbed. The MCES and IFFN Testbed evaluation approaches are also 
compared. MCES is a structured approach to evaluate Command and Control (C2) 
systems which uses standard and evolving operational research tools. The MCES 
approach provided the IFFN Joint Testbed with an air defense C2 system architecture 
which became a descriptive tool for C2 analysts to define and evaluate measures to 
determine the elTectiveness of competing air defense C2 systems. This IFFN 
application served as an evaluation and refinement of MCES as well as a tool for 
assisting the IFFN Joint Test Force in evaluating U.S. air defense C2 systems in the 
NATO area. 
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1. INTRODUCTION 



A. NATURE OF THE PROBLEM 

How much of a force multiplier can be attributed to a particular command and 
control (C“) systems? Given several alternative systems, which is best? What has 
to be measured to determine this dilTerence? Are all the relevant factors taken into 
account? These are complex questions being asked by the Joint Chiefs of Staff as 
well as other senior military commanders as they are faced with acquiring, testing, and 
operating systems. 

•A methodology is needed to describe the systems architecture which will allow 
analysts to measure systems response and attribute the effectiveness of that 
response to the elements or structural relationships which form that system. There 
is a definite need for generic tools to evaluate systems and architectures. What was 
lacking in current evaluation methodologies was a method to relate systems to 
measures of its contribution to force elTectiveness and mission accomplishment. In the 
past. evaluation has been conducted in a piecemeal fashion with assorted evaluation 
tools only for specific parts of the problem. The Joint Chiefs of Staff Command. 
Control and Communications Systems Directorate's (JCS C3S) recognized these needs 
and required development of a paradigm to evaluate competing C“ architectures 
[Ref 1: p. 8]. The Modular Command and Control Evaluation Structure (MCES) 
attempts to address this need. 

B, MCES METHODOLOGY 

MCES was developed as a structured approach to evaluate C^ systems and uses 
standard and evolving operations research tools. MCES attempts to integrate the 
previous elTorts of C^ users and analytic organizations to form a single C" evaluation 
package. 

The MCES is composed of seven separate modules which guide analysts through 
the command and control evaluation. Figure 1.1 represents the seven modules of the 
MCES methodology and the output from each module. The first module is used by 
the analyst and operational user to define the particular C^ problem. The next three 
modules set the terntinology and theory to describe the C^ system architecture which 
perntits analysts to model the C^ system and its operation. Inherent in the 
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Figure l.l MCES Modules. 

methodology is a need to describe the C system as an architecture integrating physical 
elements and process functions into a structural framework. .VICES shares common 
terminology of current systems evaluation methodologies. The MCES defines 
"architecture" as a description of an integrated set of systems whose physical entities, 
structure and functions are coherently related. The architecture provides a 
representation which will eventually lead to the ability to measure the C^ system 
response and the elfectiveness of directing forces to accomplish the mission. The C“ 
theor>' behind these modules is robust enough to allow analysts to reconfigure the 
particular C^ system physically or structurally within the architecture during the C^ 
evaluation. In module 5, measures are developed which will be used to discriminate 
between alternative configurations of the architecture. When the measures for the C“ 
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system have been identified, the sixth module requires that a suitable data generator be 
selected or developed to derive the values for the measures selected. The I'lnal .\ICES 
module is used to aggregate and evaluate the measure results in order to determine 
the optimal C“ system for a particular mission. [Ref 1: pp. 10-23] 

C. MCES EVOLUTION 

The .Modular Command and Control Evaluation Structure (MCES) was 
developed at the Naval Postgraduate School with research support from the Militarv' 
Operations Research Society (MORS) and monetarv' support from difierent military’ 
agencies [Ref 2: p. 13]. The MCES development started with military community 
research and discussion concerning the need to develop and quantify measures of 
effectiveness appropriate to C^ systems. According to the MCES principal 
investigator. Dr. Ricki Sweet, MCES is evolving as any scientific development in the 
following steps [Ref 1: p. 31]: 

1. Public discussion and mandate for clarification; 

2. Setting up the nature of the problem, the tools, definitions, and potential 
directions; 

3. First order development of the identified components; 

4. Specification of the interrelationships of the components; 

5. Testing of the theory with real problems, i.e., extra-laboratory' experiments; and 

6. Refining the structure in accordance with the test results. 

The MCES methodology is evolving in order to resolve key C^ issues. 
Throughout this evolution, C“ tools and models have been identified, developed, and 
integrated into the MCES. Having completed the first cycle of the referenced six steps 
of scientific development, MCES is now in the process of a continuing iteration of the 
last two steps of test and refinement. The Identification Friend, Foe or Neutral 
(IFFN) Joint Testbed application is an example of such a MCES test and refinement 
iteration. This scientific development will lead to a further refined, bounded and 
generic methodology that may fulfill many C^ architecture evaluation requirements. 

D. IFFN BACKGROUND 

The IFFN testbed is currently addressing the air identification problem, which is 
a subset of the overall air defense C^ problem. The testbed is representative of the 
N.ATO air defense C^ system which must operate in an environment of friendly, enemy 
and neutral aircraft to perform its air defense mission. Within the air defense C^ 
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architecture, geographically separated radars and special intelligence sources develop 
detection, track, and identification information on air objects. Computers are then 
used to store and correlate this information. Digital communications are then used to 
share the track information between command facilities. The command organizations 
develop perceptions of the battle situation and make decisions to achieve mission goals 
based on these perceptions. The command organization then implements the decisions 
by directing and controlling air defense forces to take some action against the enemy 
forces. The test concept uses a computer simulation of manned and simulated 
command centers and weapon systems employing real world operating procedures 
against varied threat scenarios. [Ref 3: pp. 3-5] 

The IFFX system bounds were defined by geographic areas of responsibility 
within the NATO Fourth Allied Tactical Air Force (4ATAF) sector. The specific 
command centers that perform the functions are: Sector Operations Centers 

(SOC), Control and Reporting Centers (CRC), Brigade Fire Direction Centers (Bde 
FDC), and Battalion Fire Direction Centers (Bn FDC). Information sources 
considered to be within the C^ system are: NATO Airborne Early Warning systems 
(NAEW), Special Information (intelligence) Systems (SIS), and other information 
sources (i.e., llight plans). The air defense C^ system architecture included the weapon 
systems when they performed C^ functions. The air defense weapon systems 
considered by the IFFN testbed are the F-15, all weather fighter, and the HAWK and 
P.ATRIOT surface-to-air missiles (SA.VIS). 

E. MCES APPLICATION TO THE IFFN TESTBED 

.A MORS workshop team specifically researched the IFFN problem during a 
conference to assist the IFFN Testbed in finding a solution to the IFFN problem. The 
initial C^ problem statement formulated by the January 1985 MORS Workshop from 
the original IFFN Testbed issues was: 

How effective is the Central Europe air defense C2 svstem in providing decision 
makers the means to assess the situation and emplov air defense assets in order 
to meet overall mission objectives? (Ref 4: p. 1] 

During the 1985 MORS C^ Evaluation Workshop, the IFFN Test Director, Colonel 
Dave .Archino issued the following challenge to the working group: 
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Develop a tool . . . specific to air defense that allows IFFN to evaluate the flow 
of C2 information throushout the C2 strticture and determine if it is useful or not 
in winning the war . . . meeting the mission objectives . . . and operational issues 
IFFN plans to address. [Ref -f; p. 1] 

It was determined that MCES could be tailored to help solve the IFFN testbed 
requirements. .Major Patrick Gandee, while a Naval Postgraduate School student, was 
the principal investigator of the MCES application to the IFFN Testbed. 

F. THESIS ORGANIZATION 

This thesis will summarize how the MCES was specifically applied to the air 
defense problem and how that application has been used by the IFFN testbed to 
address their operational issues. A comparison of the IFFN Testbed and MCES 
approaches will be presented. Since MCES was concurrently evaluated and refined in 
the IFFN application process, this thesis will continue the evolution process by 
formulating recommendations for further MCES refinement. 

The .MCES methodology will be outlined in Chapter II. The IFFN Testbed and 
its evaluation approach will be described in Chapter III. Chapter IV will describe the 
application of .MCES to the IFFN Testbed. .A discussion of the differences between 
the .MCES and IFFN Testbed approach is presented in Chapter V. Conclusions and 
recommendations concerning both the IFFN Testbed and the MCES methodology are 
presented in Chapter VI. 
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II. MCES METHODOLOGY DESCRIPTION 



A. INTRODUCTION 

The following description of MCES is taken in part from Dr. Sweet's report on 
MCES [Ref 1; pp. 10-23] and from her notes and briefings. The figures presented in 
this chapter are revised and updated versions of the ones she used in her publications 
and briefings. A more detailed description and analysis of .MCES will be presented in 
Chapter IV when the VICES application to the IFFN testbed is used as an example. 

B. MCES MODULES 

MCES is divided into seven modules which are detailed below by module. 

1. Module 1: Problem Formulation 

In module 1, the decision-maker's analysis objectives and needs are described 
for a specific C^ problem. First, the decisionmaker's needs are characterized. The 
analysts consider the decisions being formulated, assumptions about the problem, the 
level of analysis required, and the mission supported. Both the appropriate scenarios 
and assumptions underlying the evaluation are made explicit and the required level of 
analysis is determined. This problem statement is then used in the second module to 
bound the C^ system of interest. The implementation of this module results in a more 
precise statement of the problem and analysis objectives. Figure 2.1 lists the major 
actions required in module 1. [Ref 1: pp. 10-11] 

2. Module 2: C^ System Bounding 

Module 2 identifies the relevant system elements that will bound the system. 
When bounding the C system, a three component definition of a C2 system is used 
based on JCS Publication 1 [Ref 5: p. 77]. These definitions state that a C^ system 
consists of physical entities, structures, and C^ processes. Physical entities are 
equipment, software, people and their associated facilities. Structure includes 
organization, procedures, protocols, concepts of operation and information fiow 
patterns. The term "C process" refers to w'hat the system is doing or the functions 
that the process performs. Bounding the system requires bounding of the physical 
entities and structure. The processes are developed in Module 3. 

There are two issues that are raised in this module. The first issue is the 
mapping of system physical components and personnel to the systems boundaries. 
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Figure 2.1 Module 1, Problem Formulation. 

The second issue concerns determining the required levels of analysis. One method of 
graphically illustrating this process of bounding the environment is through the use of 
Dr. Sweet's "onion" as a representative of the environment. The insert of Figure 2.2 
displays this representation. Starting in the middle of the onion are the subsystems of 
the C~ system. Going out from the middle, the subsystems constitute the C2 
system. The C“ system is itself a component of the overall force which in turn is a 
part of the environment. The area outside of the "Onion Skin" is the rest of the world. 
Successive "peeling" of the "onion skin" will revel the subsystems to be evaluated. 
.Module 2 results in the bounding of the problem and the identification and 
categorization of the system elements of physical entities and organizational structure. 
Figure 2.2 depicts the activities for Module 2. [Ref 1: pp. 12-13) 

3. Module 3: C“ Process Definition 

.Module 3 defines the processes needed to fulfill the mission. The 
particular command and control process is described by analyzing the generic 
processes of the system. The proposed MCES solution to understanding the 
processes of a particular system is to use an information based paradigm similar to 
the J. Lawson C" Process Model [Ref 6: pp. 93-99] in the broader framework of 
MCES. The insert of Figure 2.3 displays a modified Lawson's generic process loop 
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Figure 2.2 Module 2, System Bounding. 
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model. One example of Lawson's generic loop model components is the ASSESS 
block. The assessments for human decisions are made by battle commanders 
performing the ASSESS function. The commanders' assessments are made by 
perceiving and assigning meaning to the overall situation. Commanders use 
information from their own sensors, feedback from their forces, and an interface with a 
separate intelligence process to develop these perceptions about the enemy and friendly 
capabilities. The other functions of Lawson's generic process model are performed 
in most processes. [Ref 1: p. 14] 

It is necessary to provide a translation of the vocabulary’ of the problem 
into the terminology of the generic process model to eflectively use MCES. This 
translation of terms helps the analyst keep from overlooking critical processes and 
provides standard vocabulary and definitions. In the past, MCES has been able to 
apply the generic C^ process model successfully with only minor modifications. There 
are different C“ process levels and interactions among the C^ process function 
components and these process relationships can become very complex. To illustrate 
different levels of processes, an example of a commander performing a decision 
function can be used. The commander passes his decision to several subordinates who 
in turn work out detailed instructions to implement that decision. These subordinates 
communicate the instructions to their forces w’hich then act in the environment. In 
command and control terms, the commander and the subordinates were performing 
separate decision functions w’ithin a process. The subordinates' decision function is 
related to the commander's decision function by the commander's decision (output) 
and the subordinates' receipt (input). In turn, the detailed instructions from the 
subordinates to the force couple the subordinate's function to the force function. This 
functional input,' output relationship forms a "structure" between separate C" process 
functions which are required to perform the mission. The structure determines the 
information flow. [Ref. 1: p. 15] 

In a distributed system, processes may be related to other processes. The 
processes that have been valuable to MCES evaluations for describing distributed 
information flow are [Ref. 1: pp. 70-73]: 

1. Intelligence (INTELL) Process. Assigns meaning .to observed activities and 
situations and forecasts changes in the current situation. 

2. Crosstell (XTELL) Process. A subset of the communications process, which 
provides for sharing of information throughout the C2 system to support 
decisions and their implementation. 

3. Execution Level C2 Process. Directly controls weapon systems. 



18 




Figure 2.3 Module 3, C“ Process Definition. 
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A complete picture of the C” systems architecture includes the intelligence process and 
how it interfaces with the process. XTELL is the sharing of information and is 
needed to describe the coordination between distributed systems. At a single 
conimand node, XTELL is simply a process function but on a systems level it is a 
networking process. At a single node, the process directly controls weapon 
systems. .At a higher level, the C“ process coordinates the directing of the weapon 
systems. These processes are dynamic descriptions of what the system is doing. 
XTELL and INTELL processes interfaced with the process in a distributed system 
are ultimately linked to the weapon systems which perform the mission. 

Dr. Sweet emphasizes that this module forces the analysts attention on 
(Ref 1: p. 14); 

1. the environmental cause or initiator (i.e., the enemy force) of the C2 process 
that results Irom a change in the desired state; 

2. the internal C2 process functions that characterize what the svstem is doing 
such as sense, assess, generate, select, plan, and direct; and 

3. the input and output from the internal C2 process that couples with the force 
process 

Figure 2.3 represents the major actions required in module 3. As a result of the 
implementation of this module, the functions of the process for a given problem are 
identified and mapped to the generic process loop. 

4. Module 4: Integration of System Elements and Functions 

Module 4 relates the information flow' to a system by means of its C“ 
process functions. Functions are subsets of the process and represent what the C2 
system actually does or accomplishes. The relationship of the C“ physical entities to 
the process functions and organizational structure is also formulated in this module. 
This integration is accomplished by making e.xplicit the relationships between these 
components. Figure 2.4 outlines the actions required in Module 4. 

The first step is to map the physical entities of man and machine which 
perform the functions and produce output to an organizational structure. .All 
functions can be potentially performed in a single node or be distributed between 
dilTerent nodes so this mapping results in an organizational structure which graphically 
depicts a single node or a distribution of command and weapon nodes depending on 
the system's unique configuration. 

Next, the flow of information is charted by techniques such as Data Flow- 
diagrams (DFDs) [Ref. 7: pp. 99-1 15] or Petri Nets [Ref 8] which may be used to 
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Figure 2.4 Module 4, Integration of System Elements and Functions. 

describe information flow through the process model. DFDs and Petri Nets will be 
discussed in greater detail in Chapters V and VI. DFDs are an example of a technique 
that has been productive in this module in determining relationships between processes 
and information How. Data Flow Diagrams (DFDs) are lirst constructed to show 
information How through the process model. The inputs and outputs of each 
function are determined and related to the other functions in the process. In the next 
step, a transform analysis is performed to uncover the transform center (process center) 
and to deterntine the subordinate and superordinate relationships between the 
transform center and the individual C" functions. Information Hows into the 
transform center and control information flows out of the transform center. These 
input, output relationships describe the internal information flow between separate 
process functions. The data flowing into the transform center or process center is 
information flow while the data flow out of the transform center is control information 
flow in the form of action requests or commands. Thus a hierarchical "structure" has 
been defined in terms of the mission essential information flow between functions 
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within the C" process. From this information, a C“ system architecture will eventallv 
be formulated. These procedures will result in the description of how the elements and 
players function together, which is basically relating information flow to organizational 
structure. (Ref 1: pp. 16-17] 

5 . Module 5: Specification of Measures 

.Module 5 specifies the measures necessary to evaluate the C“ problem. These 
measures are then classified as to their level of measurement, i.e., dimensional 
parameters, measures of performance (MOPs), measures of effectiveness (MOEs), or 
measures of force effectiveness (MOFEs). MOFEs are used to describe the actions 
between the force and the environment. When the C system is combined with the 
force, the environment will be effected and MOFEs measure this force effect. Within 
the force boundarv, .MOEs are used to measure how the combat force is effected bv the 
C“ operation. MOPs are applied at the system boundary and measure how well the 
C'^ system performs its functions. For the subsystem within the boundary of the 
system, dimensional parameters are used to measure the limits of the subsystems. The 
resulting measures may be used to determine differences in a 0“^ system when utilizing 
alternative configurations of the physical entities, structure, or processes. Figure 2.5 
graphically depicts the "onion" with its corresponding measures and the actions 
required in Module 5. 

This MCES implementation results in the specification of a set of measures 
focused primarily on the process functions. The process functions identified may be 
used to derive a complete set of relevant measures which can then be subjected to 
further scrutiny. A set of measures can be compared to a set of desired measures 
characteristics as shown in Table 1 [Ref 1: p. 20] to insure that the measures are 
useable. 

6 . Module 6: Data Generation 

In Module 5, one of several types of data generators such as exercises, 
experiments, simulations, subjective judgements, or relevant experiences is selected to 
generate the necessary values for the measures formulated. These values may be either 
measured directly or indirectly. The analysts consider the reproduciblity of results, 
precision and accuracy, timing of collection, environmental controls, and experimental 
design during this module. A timeline is formulated to set the completion dates for the 
data generation phases. Using the designated data generator, the resulting values for 
these measures constitute the output of this module. Figure 2.6 outlines the data 
generation module. [Ref 1: p. 21] 
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Figure 2.5 Module 5, Specification of Measures. 
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TABLE 1 


CRITERIA FOR EVALUATION MEASURES 


CHARACTERISTICS 


DEFINITION 


Mission oriented 


Relates to force/system mission. 


Discriminatory 


Identifies real difference between 
alternatives. 


Measurable 


Can be computed or estimated. 


Quantitative 


Can be assigned numbers or ranked. 


Realistic 


Relates realisticly to the C^ 
system and associated uncertainties. 


Obj ective 


Can be defined or derived, 
independent of subjective opinion. 


Appropriate 


Relates to acceptable standards and 
analysis objectives. 


Sensitive 


Reflects changes in system variables. 


Inclusive 


Reflects those standards required 
by the analysis objectives. 


Independent 


Is mutually exclusive with respect 
to other measures. 


Simple 


Is easily understood by the user. 



7. Module 7: Aggregation of Measures 

Module 7 is the final module and addresses the issue of the aggregation and 
interpretation of the observed values of the measures. Figure 2.7 depicts the 
aggregation and interpretation process. From data generation, values for the identified 
measures will be obtained and analyzed. One of the analysts' concerns will be to relate 
conunand and control systems to some measure of force effectiveness which is 
sometimes termed the force multiplier effect. For MOFEs, the intent of aggregation is 
to relate the system with combat systems to indicate combat outcomes. After 
aggregation, the issues of measure causality, sufficiency, and independence are to be 
considered. Scenario dependence must also be addressed. Because combat is ver>’ 
complex, many measures will not show significant differences. .Analysis must be 
conducted to determine the important factors in the particular scenarios. .At this 
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Figure 2.6 Module 6, Data Generation. 

crucial point, the analysts must decide if their questions can he answered by their 
analysis. Credibility and reliability are major concerns to the decisionmakers. 
IRcf. 1: pp. 21,221 

C. COURSES OF ACTION 

'1 he results of the MCFS iteration are provided to the decisionmaker. Figure 2.S 
represents the actions and results of each iteration of the .MCFS modules. .At least two 
courses of action are then available to the decisionmaker based on the results. 1 he 
decisionmaker may directly implement the results of the MCES evaluation or he may 
identify the need for further study or require another iteration of the .MCFS analysis, 
fhe decisionmaker may interact with the MCES analysis ellbrt to further guide the 
analysts by identifying errors in assumptions, clarifying the bounding, etc. 1 he analysis 
could be modified by infusing new directions or objectives based upon the results of 
previous .MCES modules completed. I'or e.xample, the bounding of the C" system may 
generate the observation that signillcant interfaces are outside the originally conceived 
scope of the study and require a return to the problem formulation module. 
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Figure 2.7 Module 7, Aggregation of Measures and Interpretation. 

D. USES 

MCFS can assist in the areas of management and analysis. MCES assists in 
the systematic specification of the problem by focusing on identified essential 
characteristics of the C" system. It permits a senior analyst to conduct a C“ 
evaluation effectively. .MCES assists the analyst in forming a concise conclusion and 
provides the manager with supporting data for decisionmaking. I he I FEN I'estbed is 
a good example of a C" evaluation of air defense and will he illustrated in the following 
chapters. 
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Figure 2.8 MCES Modules and Output. 
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III. IFFN TESTBED DESCRIPTION 



A. THE IDENTIFICATION PROBLEM 

The Identification Friend. Foe or Neutral (IFFN) Testbed, comprised of a U.S. 
Army and Air Force Joint Test Force at Kirtland Air Force Base, is investigating ways 
to enhance the identification of friendly, neutral, and enemy aircraft. A realistic 
scenario used by the IFFN Joint Test Force forecasts that in future air battles our 
tactical air defenses will be faced with sophisticated enemy fighters capable of engaging 
our forces with beyond visual range (BVR) weapons The large number of enemy 
aircraft with their use of low level tactics, high speeds, and Electronic Countermeasures 
(EC.M) challenges our air defense systems. The air defense system must be able to 
identify and characterize the enemy attack and then direct sufficient force in time to 
neutralize it. (Ref 9: p. 47] 

Effective performance of the active air defense mission requires a capability to 
correctly identify aircraft in a timely manner in order to facilitate the air defense 
weapon system's ability to employ its weapons. This air defense process is done 
through a complex arrangement of personnel, equipment, communications facilities 
and procedures which form a C^ system. This requirement is particularly important in 
the European theater where large numbers of friendly, neutral, and enemy aircraft will 
be part of the tactical air environment. In this environment, surface-to-air and air-to- 
air weapon systems must operate in conditions beyond ranges where positive visual 
identification can not be performed. The problem is aggravated when modern 
electronic warfare (EW) threats, particularly those of the Warsaw Pact are considered. 
Numerous studies have revealed that current electronic identification capabilities have 
numerous problems including being too slow, poor at positively identifying enemies 
and friends, insufficient range capability, and being subject to interference and 
electronic countermeasures (ECM). The inability of air defense weapon systems 
operators to accurately and rapidly discriminate friend, foe. or neutral aircraft results in 
the ineffective use of these weapon systems. These problems have stimulated activity 
within the US military and NATO to develop an effective NATO identification system. 
The IFFN Testbed is a partial attempt by the United States at solving this air defense 
problem. [Ref 3: p. 1] 
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The function of acquiring, correlating, fusing, and dissentinating direct and 
indirect IFFN information is an important part of air defense C^. This IFFN 
information serves as a basis for threat evaluation and engagement control. It is also 
the function of air defense command and control to provide sufficient identity 
information, bearing, and range to the allocated weapon system so that the weapon 
system is able to acquire and engage a target with a high degree of confidence. 
Acquisition and engagement information processed in a timely manner will permit 
effective weapons employment. .An IFFN system should also provide the necessary 
information to allow passage of friendly and neutral targets. 

B. THE AIR DEFENSE PROCESS 

Identification is an integral step in the air defense process and begins with 
tasking and disposition of assets for air surveillance. The process continues through 
detection of an intruding target and ends with a decision to engage the target with an 
air defense weapons system. This process is characterized by complex relationships 
between air surveillance, C“, and weapons systems. The targets will be identified as 
friendly, neutral, hostile, or unknown. The hostile targets will be given different 
priorities for engagement or may be engaged by other air defense assets. Air defense 
C“ must have sufficient identity information to evaluate the intent of hostile targets 
and assess the threat level posed by those individual targets. Air defense must then 
prioritize those targets for engagement, allocate the targets for engagement by specific 
weapons systems, and aid a weapon system in acquiring that target without 
endangering other targets. This must be done in a constantly changing air 
environment where identity determination is a dynamic process involving people, 
hardware, and software. 

The classical sequence of air defense is detect, identify, engage, and destroy. This 
classical sequence used by the IFFN Testbed is a simplification built from a number of 
decisions and functions performed separately or mutually by elements and air 
defense weapon systems. Most of these functions and decisions are dependent upon 
identification. Complicating this simple sequence is the fact that identification is both 
a process and a decision. As a process, identification is a constant gathering and 
correlating of information about a potential target from all sources of direct and 
indirect identification information. This process is continuous up to the final 
disposition of a target by the air defense system. As part of the identification process. 
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the C" system must correlate information from all sources and resolve any conllicts. As 
a decision, an identification must be made before further action can occur, either 
actively or by default. This decision is influenced by all the sensors available to the air 
defense system, by the background intelligence on the air situation, and by the 
operational procedures such as rules of engagement and weapons control status. The 
decision-maker must either make or delegate the identification decision prior to the 
engagement decision. [Ref 3: p. 8] 

When performing the classical air defense sequence within an integrated air 
defense system, the identification decision is also part of a larger process which is 
performed by both C" elements and air defense weapons systems. The process 
becomes more complex when additional sources of direct and indirect information are 
available and higher levels of command and control participate. Individual weapon 
systems are simultaneously performing detection, tracking, and identification as part of 
their target acquisition function. They can be aided in performing this function by the 
contmand and control system as it exercises its function of engagement control. When 
operating autonomously, weapon systems are limited to their organic detection, 
tracking, and identification capability. The detection, tracking, and identification of 
the entire system can be better used for target allocation and acquisition when the 
command and control system provides engagement control in a centralized mode of 
operation. Identification is a major factor in the performance of weapon systems to 
defeat the enemy. 

C. IFFN JOINT TESTBED CONCEPT 

The Department of Defense's proposed partial solution to the N.ATO air defense 
problem was the development of the IFFN Joint Testbed to gather analytic data on 
the problem so that solutions could be formulated. The testbed uill assess baseline US 
capabilities within the NATO air defense system to perform the IFFN function, 
identify deficiencies in the performance of that function, and propose potential near- 
term procedural and equipment modifications for further testing. One issue that will be 
addressed by the IFFN Testbed is the indirect information process and how its use 
may improve the performance of air defense systems to aid C^ and weapon systems 
nodes. A testbed approach was taken so that a number of diflerent C^ strategies could 
be tested without having to actually use the real equipment and weapons. The testbed 
was envisioned to simulate as close as possible the real threat and the U.S. equipment 
and procedures used in N.ATO. [Ref 3: pp. 1,2] 
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The level of complexity of the air defense system is enormous, thus, the IFFX 
Test Force expended a large amount of effort in understanding general C” systems 
before modeling the NATO air defense system. A simulation testbed was ultimately 
chosen to evaluate the alternative air defense C" systems. The IFFX Test Force is 
attempting to determine air defense identification measures of performance (MOPs) 
and measures of eflectiveness (MOEs) that will lead to the evaluation of the utility of 
the different configurations of the air defense architecture. 

I. Baseline Architecture 

The IFFX baseline architecture was formulated as a combination of hardware, 
software, procedures, and doctrine that is planned to exist in the late 1980's. The 
criteria will also be subjected to the projected Warsaw Pact 1987-1990 threat and is 
consistent with Defense Intelligence Agency estimates of enemy capabilities and orders 
of battle for that period. The timeframe chosen w'as a compromise between possible 
near-term benefits and results which will have long range applicability. Certain 
modifications that will be fielded in 1987 or beyond will be candidates for follow-on 
tests using the IFFX testbed. [Ref 3; p. 3] 

The testbed geographical area of interest is the XATO environment in which 
US Army and .Air Force units operate jointly and under the control of associated 
elements of the XATO Air Defense Ground Environment (XADGE) System. The 
IFFX testbed will focus on the battle management area of a representative X.ATO 
Control and Reporting Centers (CRC) located in the Forth Allied Tactical Air Force 
(4AT.AF) area. Representation of key associated XATO command and control nodes 
and information sources is required in the IFFX testbed. Figure 3.1 depicts the key 
components of the IFFX Testbed. [Ref 3; pp. 3-6] 

a. Command and Control Units 

Command and control units are those representative units which direct or 
control the beyond visual range (BVR) weapon systems and execute the active air 
defense mission. The specific command centers that perform these C functions are: 
Sector Operations Center (SOC), Control and Reporting Centers (CRC), Brigade Fire 
Direction Centers (Bde FDC), and Battalion Fire Direction Centers (Bn FDC). 

b. Information Sources 

Sources which provide information for identification, target allocation, and 
target acquisition in the air environment to the C^ units and w'eapon systems are 
categorized as information sources. These are: XATO Airborne Early Warning 
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Figure 3.1 IFFN Air Defence Structure. 



systems (X.-XE^V). Special Information (intelligence) Systems (SIS), and other 
information sources (i.e.. Eight plans). 
c. lyeapon Systems 

A’eapon systems are those representative B\'R weapon systems winch are 
operated by L'S forces. For the IFFN Testbed, the F-15. HAWK, and P.\l RIOT have 
been selected as the representative BVR weapon systems. Due to resource constraints 
on the IFFX' Testbed, only these three weapon systems will be used. 

2. Major Operational Issues 

The testbed will generate and record the data necessary for analysis and 
recommendations on IFFX' issues There are three major operational issues considered 
by the IFFN Testbed which are listed below. 
a. Issue I 

"What is the contribution of indirect identification information to the 
ability of L’S air defense command and control systems operating in NATO to 
correctly identify airborne targets, to use identification in performing target allocation, 
and to aid subordinate air defense weapon systems in perfornung target acquisition?" 








[Ref. 10; p. 2|. Indirect ID information is that ID information that is not direct 
electronic IFF returns. Indirect information usually refers to all other ID information 
that arrives as a facility such as flight plans, area of operations, intelligence reports, 
etc. However, direct ID information once passed to another facility also becomes 
indirect ID information. Satisfying the first major operational issue will provide a 
baseline assessment of the expected identification performance of a representative 
NATO air defense system operating in the 4ATAF area. Studying the first issue will 
also provide a fuller understanding of the relationship between the identification 
performance of the system and the performance of the overall active air defense 
mission. 

b. Issue 2 

"What are the deficiencies in air defense system use (collection, formation, 
dissemination, and use) of indirect identification information for which solutions are 
not currently planned?" [Ref 10: p. 2]. The second major operational issue will 
identify weaknesses in the identification process and allow for a qualitative comparison 
of those weaknesses identified during testing. Potential corrective actions for these 
identified deficiencies can then be developed. These recommended corrective actions 
could take the form of changes in doctrine, procedures, system software, 
communications connectivity, or addition of new data sources, or various combinations 
of these solutions. 

c. Issue 3 

"What near-term procedural and equipment modifications should be 
recommended to overcome deficiencies?" [Ref 10: p. 2]. This issue will address 
productive near-term solutions to the IFFN problem. 

3. Hybrid Approach 

Two major options were considered during feasibility studies when developing 
the test concept. Field exercises and computer-based simulation were both evaluated 
and compared. A hybrid approach was ultimately selected which permits the use of the 
best of both options. The concept is centered around live operators using actual 
tactical hardware and accepted simulations of hardware and software called Live 
Participating Units (LPUs). Real-time computer models stimulate the LPUs as well as 
represent the background workload for these units. This man-in-the-loop simulation 
will be carried out through all the tests. Since the IFFN Joint Test Force developed a 
hybrid simulation testbed composed of simulated participating units, there was no 
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requirement for live operation of aircraft, weapons, radars, nor field deployment of 
weapons and command and control systems. To implement this test concept a 
distributed testbed was established at a central facility at Kirtland Air Force Base. New 
Mexico. The testbed at Kirtland generates and distributes the tactical scenario, 
controls test execution, and monitors the response of geographically distributed Patriot 
and Hawk LPUs at the ,'\rmy Air Defense Center at Fort Bliss, Texas. 

A realistic scenario environment was necessary to ensure realistic and accurate 
results. The combination of a few high fidelity LPUs responding to simulated and 
manned units was determined to be an excellent method to simulate the 
interdependency and interaction of air defense units. The IFFN Test Force conducted 
a substantial testbed certification elTort with joint service participation to validate and 
calibrate testbed performance against joint and combined field exercises for the first 
test series. Further credibility and validity tests will be made after each test series. 

4. Models 

The IFFN models that were developed can be categorized as interactive or 
noninteractive. The interactive models react dynamically to perceived changes in the 
air battle situation. They may receive inputs such as data link messages from the other 
models or LPUs and may initiate messages in response to this input or their own. The 
output of these models is dependent on the specific dynamics of the air battle. The 
applications of the interactive models for the IFFN testbed are: sensor models, missile 
models, and dynamically controlled aircraft models. Examples include radar, electronic 
IFF, Patriot SA.Ms. and fighter and attack aircraft platforms. [Ref 10: p. 10] 

The IFFN noninteractive models do not react to the air battle dynamics. 
They are less complex models and simply generate selected messages and actions at 
preprogrammed times according to a script prepared prior to the test. The models are 
considered suitable for simulating those facilities that do not dynamically interact with 
the identification process, but do provide orders, procedures, and other information on 
a one-way basis such as certain higher echelon planning facilities. N’oninteractive 
models will also be used to simulate aircraft following programmed flight profiles which 
are not automatically reactive to the air battle environment. The Sector Operations 
Center (SOC) is an example of a noninteractive model used in the IFFN Testbed 
simulation. Examples of weapon platforms are transport, patrol, and tanker aircraft. 
[Ref 10: p. 14] 
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D. TESTBED PHASES 

In order to minimize technical and program risks, a phased testbed acquisition 
was adopted. The test approach is based on seven test series. The series will consist 
of weapons systems, command and control systems, and associated data links. One of 
eight planned phased simulations has been completed. The following is a description 
and list of the systems involved. [Ref 10: pp. 3.4] 

1. Test Series 1 

Series 1 tests the identification performance of a representative US Army 
Surface-to-Air Missile (SAM) System which is the PATRIOT fire unit. The systems 
tested are; 

1. PATRIOT Fire Unit (FU), and 

2. P.ATRIOT Air Defense Information Language (PADIL). 

2. Test Series 2 

Series 2 adds the PATRIOT'S first echelon of command and control, the 
Battalion Fire Direction Center. The systems tested are; 

1. P.ATRIOT FU, 

2. PATRIOT Battalion Fire Direction Center (Bn FDC), and 

3. PADIL. 

3. Test Series 3 

Series 3 adds the next level of C^, the Brigade Fire Direction Center. The 
systems tested are: 

1. P.ATRIOT FU, 

2. PATRIOT Bn FDC. 

3. PATRIOT Brigade Fire Direction Center (Bde FDC), 

4. P.ADIL, and the 

5. .Army Tactical Data Link-1 (ATDIL 1). 

4. Test Series 4 

Series 4 tests only the USAF's fighter-intercepters, the F-15 . The system 
tested is the F-15 "Eagle" Intercepted 

5. Test Series 5 

Series 5 adds associated USAF C^ nodes and information sources to the F-15 
system. The systems tested are: 

1. F-15, 

2. USAF Control and Reporting Post/Message Processing Center (CRP MPC), 

3. NATO Airborne Early Warning System (NE-3A), 
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4. Special Information System (SIS), 

5. Tactical Digital Information Link - A (TADIL-A), and 

6. TADIL-B. 

6. Test Series 6 

Series 6 will integrate the Army systems from Series 1-3 with the L'SAF 
systems from Series 4 and 5. This will now be a joint operations test. The systems 
tested are; 



1. 


PATRIOT FU, 


2. 


PATRIOT Bn FDC, 


j. 


PATRIOT Bde FDC, 


4. 


F-15, 


5. 


NE-3A. 


6. 


CRP, 


7. 


SIS, 


8. 


TADIL-A, 


9. 


TADIL-B, 


10. 


PADIL, 


11. 


ATDL-1, and 


12. 


NATO Link-1. 




7. Test Series 7 



Series 7 will add a CRC to form the total system to be tested. The systems to 
be tested are; 

1. PATRIOT FU, 

2. PATRIOT Bn FDC, 

3. PATRIOT Bde FDC, 

4. F-15, 

5. NE-3A, 

6. CRP, 

7. SIS, 

S. NATO Control and Reporting Center (CRC), 

9. TADIL-A, 

10. TADIL-B, 

11. PADIL, 

12. ATDL-l,and 

13. NATO Link-1. 
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E. IFFN TEST CELL MATRIX 

The simulation will be conducted using a controlled variable approach. Different 
simulation test cells are used in which some variables are held constant while others are 
left to fluctuate and eventually lead to the determination of the variables' impact on 
the system. The basic test structure considers both a fully integrated air defense 
system and several subsets of the system. The different configurations allow different 
variables to be isolated to establish their contribution to the particular air defense 
system. Each trial will be run with a constant measurement volume of a prescribed 
radius with only predetermined variables changed. The measurement volume is the 
area covered by the air defense unit or weapon system. The test cells of the matrix are 
used to generate comparative data under various environmental conditions. Figure 3.2 
represents the basic test matrix of ten test cells used by the IFFN testbed. 

Data items are collected from collection messages that are generated by data 
events during the simulation. Data events are events that take place during the 
simulation such as information arrival and actions performed by the air defense 
system nodes. For Test Series 2, there are fourteen data events that are collected from 
each test cell simulation to be used in the calculation of the MOEs and MOPs with 
other data events used for deficiency analysis. All MOEs and MOPs are divided into 
two groups of probability measures and distribution measures. These measures are not 
necessarily measures of the variables themselves but are intended to be measures of the 
results of the variables impact on the system. [Ref 11: p. 20] 

F. MEASURES 

General categories of measures were sought to derive values that would 
eventually lead to discrimination among the different C systems. The simulation can 
not completely characterize the performance of the fully deployed air defense systems, 
so absolute conclusions about the performance of the air defense systems are nearly 
impossible. With this shortcoming in mind, measures of the relative change in the 
performance effect of the variable under varying conditions will be used when only 
large and significant differences are noted. This means that a low confidence level will 
be used w’hen analyzing the data. Figure 3.3 depicts the general approach taken by the 
IFFN Testbed in resolving its IFFN issues. 
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N. SCENARIO 
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• NOM (Nominal) 

• ACP's (Airspace Control Procedures) 

• Q&A IFF (Question and Answer Electronic IFF) 

• FU (Fire Unit) 

• ilD (indirect Identification) 

• AUTO ( Autonomous) 



Figure 3.2 IFFN Basic Test .Matrix. 
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Figure 3.3 IFFN Test Design .Approach. 



1. Test Series 2 Objectives 

There were originally si.\ objectives formulated from the three issues for Test 
Series 2 that eventually led to the formulation of the IFFN Testbed measures. These 
objectives are listed below without the more detailed sub-objectives for each objective. 

1. Objective 1 - Assess and contrast the performance of the PATRIOT battalion 
under centralized and decentralized control. 

2. Objective 2 - .Assess the impact of chansine and removine Airspace Control 
Procedures (.ACPs) on the operational perTormance of tTie centralized and 
decentralized battalion and autonomous fire unit. 

3. Objective 3 - Determine the value and impact of perfect ID and communication 
system performance on PATRIOT battalion performance. 

4. Objective 4 - Evaluate the impact of various changes in direct and indirect ID 
performance and interaction. 

5. Objective 5 - .Assess the influence of the fire unit ID function on the ablitv of 
the PATRIOT battalion to perform its lunctions. 

6. Objective 6 - Identify and subjectively evaluate any PATRIOT operational 
deficiencies noted during testing. (Ref II: pp. 2. 1,2.2] 
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2. Measures of Effectiveness (MOE) 

There are three primary’ MOE areas formulated for all the test series. The 
IFFN Testbed originally identified three functions that were performed in the air 
defense process: identification, target allocation, and target acquisition. [Ref 3: pp. 
17.1S] 

a. MOEs for the Identification Function 

These .MOEs will describe how well the weapons and systems are able 
to identify or recognize airborne objects and assign them to appropriate identification 
categories. 

b. MOEs for the Target Allocation Function 

These MOEs will relate identification information used by systems to 
allocate air defense weapons against hostile aircraft and prevention of misallocation of 
weapons against friendly aircraft. 

c. MOEs for the Target Acquisition Function 

These MOEs provide the measures relating indirect identification 
information to the weapons systems. Target acquisition provided by the command and 
control structure to the weapons systems is a part of these MOEs. 

d. MOE List 

The MOEs for Test Series 2 that measure the needed information for the 
evaluation issues are listed in Table 2 [Ref 11: p. D25j. The letter "P" identifies 
probability measures while the letter "R" signifies range distributions and the letter "T" 
represents time distributions. 

3. Measures of Performance (MOP) 

Measures of Performance (MOP) for the IFFN Testbed were submeasures of 
the air defense functions and therefore subsets of the MOEs. An example is the 
probability that a passed ID is correct which is a submeasure of MOE 3, probability of 
identification of an aircraft. The MOPs for Test Series 2 that were determined to 
measure the needed quantities or qualities to resolve the six stated objectives are listed 
in Table 3 [Ref II: pp. D23,D24] with corresponding MOE number references. 

4. Measures of Force Effectiveness (^tOFE) 

Measures of Force Effectiveness are global measures that determine how 
effectively the mission is accomplished. Target hits and damage assessments were not 
modeled in this testbed so MOFEs could not be used. 
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TABLE 2 






MOE DEFINITIONS 


MOE# 


MOE 


MOE DEFINITION 


1. 


P(D/V) 


Probability of detecting an aircraft 
given that it has entered a system 
measurement volume. 


2. 


P( T/D) 


Probability of tracking an aircraft, 
given that it has entered a system s 
measurement volume and has been detected. 


3. 


P( I/T) 


Probability of identification of an 
aircraft, given that it has entered 
a system s measurement volume and has 
been tracked. 


4. 


P(A/I) 


Probability of allocation of an 
aircraft- given that it has entered 
a system s measurement volume and has 
been identified. 


5. 


P(E/A) 


Probability of engagement of an 
aircraft, given that it has entered 
a system s measurement volume and 
has been allocated. 


6. 


P(E/V) 


Probability of engaging an aircraft. 


7. 


P(E/F) 


Probability of engaging a friedly 
aircraft. 


8. 


P( E/N) 


Probability of engaging a neutral 
aircraft. 


9. 


P(E/H) 


Probability of engaging a hostile 
aircraft. 


10. 


T( Tot) 


Distribution of times elapsed between 
detection and engagement. 


11. 


P( EBO) 


Probability of engaging a hostile 
aircraft before it fires 
a missile or drops ordnance. 


12. 


R( FE) 


Distribution of ranges from aircraft to 
FSCL at time of engagement. 
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TABLE 3 

MOP DEFINITIONS 



T 



MOE 

ref# 


MOP 


MOP DEFINITION 


1) 


R(D) 


Distribution of ranges from aircraft 
to detection unit at time of detection. 


1) 


R( FD) 


Distribution of ranges from aircraft to 
FSCL at time of detection. 


2) 


R(T) 


Distribution of ranges from aircraft to 
tracking unit at the time the track was 
established. 


2) 


R( FT) 


Distribution of ranges from aircraft to 
FSCL at time of tracking. 


2) 


T(DT) 


Distribution of times elapsed between 
detection and tracking. 


3) 


P( IX/YI ) 


Probabilities of identifying an aircraft 
as category X ( friend< neutral, or 
hostile), given that its true identity 
is Y (friend, neutral, or hostile) and 
that It has been detected. 


3) 


R( I) 


Distribution of ranges from aircraft to 
identifying unit at time of ID. 


3) 


R(FI) 


Distribution of ranges from aircraft to 
FSCL at time of ID. 


3) 


T( TI ) 


Distribution of times elapsed between 
tracking and ID. 


3) 


P(Pass) 


Probability that a passed ID is correct. 


3) 


P( Res) 


Probability that an ID conflict is 
resolved while the aircraft is still in 
the weapon system s measurement volume. 


3) 


T( Trans ) 


Distribution of times elapsed between 
receipt and retransmission of ID 
information by a C2 node. 


3) 


P( Amp) 


Probability that an ID includes track 
amplification information. 


4) 


P( A/YI) 


Probabilities of allocating an aircraft, 
given that its true identity is Y 
(friend, neutral, or hostile) and that 
It has been identified. 


4) 


R(A) 


Distribution of ranges from aircraft to 
allocated unit at time of allocation. 


4) 


R( FA) 


Distribution of ranges from aircraft to 



FSCL at time of allocation. 
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TABLE 3 






MOP DEFINITIONS (CONT'D. ) 


4) 


T( lA) 


Distribution of times elapsed between ID 
and allocation. 


5) 


P( E/YA) 


Probabilities of engaging an aircraft, 
given that its true identity is Y 
ffriend, neutral, or hostile) and 
that it has been allocated. 


5) 


R(E) 


Distribution of ranges from aircraft to 
engaging unit at time of engagement. 


5) 


T( AE) 


Distribution of times elapsed between 
allocating and engagement. 



G. ANALYSIS OF DATA 

1. Exploratory’ Data Screening 

The test data results are first examined and screened for outliers and to verify 
underlying assumptions such as normality, independence, constant variances, and zero 
mean. Various methods are used to statistically check the data. Data screening 
involves a number of methods listed below. [Ref 11: pp. D37-D42] 

a. Box and line plots 

Box and line plots are used for time and range distributions. The box or 
line plot allows pictorial presentation of a set of distribution measures for a set of trials 
or number of test cells. 

b. Frequency distributions (Histograms) 

Histograms are used also for time and range distributions. These show 
visual evidence of normality as well as extreme outlying values. 

c. Scatterplots 

Scatterplots are used for time versus range plots distributions. Bivariate 
scatterplots provide a visualization of the relationship between two continuous 
variables. 

d. Normal probability plots 

Probability plots are used to determine probability types and fits. Normal 
probability plots provide visual ev’idence of the difference between a given distribution 
and a Gaussian distribution. 
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2. Data Analysis 

The data analysis that follows screening is listed below. 

a. Paired T Tests 

Means are tested using this test. The assumptions concerning the 
underlying populations are that they are independent samples and the variances are the 
same. If they are not, then F tests are used. 

b. Analysis of Variance (ANOVA) 

.ANOV.A allows inferences to be formulated about differences in "treatment 
elTects" brought about by the test variables which are controlled from cell to cell. 
Inferences are made by estimating how much of the variability in test data is explained 
by the effect of the test variables and how much is due to random error. 

c. Hypothesis Testing 

The .ANOV.A provides a basis for the formal hypothesis test that the trial; 
cell means or specific subgroup means are all equal. 

d. Contingency Table Analysis 

This analysis is sometimes called row by column (RC) table analysis. This 
test is used when more than two outcomes are possible, a frequent occurrence among 
the test cells. Each 2x2 contingency table analysis will be performed for comparing 
probability measures from cell to cell on as many of the measures as practicable. 

e. Regression 

Regression analysis determines the statistical relationship between one or 
more independent variables and a dependent variable. Curve fitting is accomplished by 
regression of the dependent variable on the independent variable. 

f. Correlative analysis 

The relationship between the independent and dependent variables or 
causality is determined by correlation analysis. This analysis determines what 
proportion of the variation of the dependent variables can be attributed to the 
relationship with the independent variable. 

g. Standard Normal Theory Approximations 

These series of tests are used to determine what type of probability 
distribution exists and how good that data fits that distribution. 

h. Deficiency analysis 

.After the data has been checked and analyzed, the causes of the differences 
in the configurations are proposed and examined. The objective is to find the 
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underlying cause or reason behind the difierences. This could be helpful to 
decisionmakers in allocating limited resources to dilTerent C“ configurations. 
[Ref. 11: pp. D11-D29] 

H. IFFN PROGRESS SUMMARY 

It is evident that the IFFN Testbed has made progress in its attempt to evaluate 
the NATO air defense system. The IFFN air defense problem is definitely complex 
and the IFFN Testbed has understandably committed large amounts of resources to 
the problem. The Testbed is a good concept for an experimental design to test 
competing systems since all of the alternative systems can not be tested using actual 
equipment due to resource constraints or present configuration limitations. The 
IFFN Test Force has been careful to insure that the testbed is credible. Only Test 
Series 1 has been completed with Test Series 2 to begin in March 1987. 
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IV. PROPOSED MCES APPLICATION TO THE IFFN TESTBED 



A. INTRODUCTION 

Priman' research into the application of MCES as an evaluation tool for the 
IFFN Testbed was undertaken by Major Patrick Gandee, U.S. Air Force, when he was 
a Naval Postgraduate School student. Two .Military Operational Research Society 
(.MORS) teams also contributed to the proposed MCES application during two MORS 
conferences. The bulk of Major Gandee's and the MORS teams' work with the IFFN 
Testbed is included in Major Gandee's thesis [Ref 12] and later refinements by Major 
Gandee as a staff member of the Joint Chiefs of Staff Command and Control Systems 
Directorate [Ref 9; pp. 49-58]. Dr. Ricki Sweet, who was Major Gandee's thesis 
advisor, provided the then current MCES methodology guidance. This application 
study was also supported by the Naval Postgraduate School and the Office of the Joint 
Chiefs of Staff. Major Gandee received assistance from IFFN Testbed personnel 
including Colonel Dave Archino, Director, and Major Mike Grey, Chief Operational 
.Analysis Section. These IFFN Testbed personnel also participated in the MCES 
application to the IFFN Testbed. The following application is mainly taken from 
Major Gandee's thesis, notes, and briefings along with Major Grey's notes and 
conversations plus research of IFFN Test Force test plans. 

B. PROBLEM FORMULATION 

Utilizing the first .MCES module, the MORS team and .Major Gandee 
reformulated the IFFN problem which was a subset of the air defense C^ process 
problem. Within air defense C^, the emphasis was on allocating multiple hostile 
targets to weapons systems for engagement. .Major Gandee understood that this air 
defense C^ process description should consist of a complete set of battle management 
functions which were needed to direct the weapon systems to perform the air defense 
mission. Other issues considered by Major Gandee in the first module were the 
different evaluation levels and analysis objectives. Issues such as procedural control 
and centralized and decentralized control were also researched and reviewed. Figure 
4.1 lists the major actions taken by .Major Gandee in applying .VICES to the IFFN 
Testbed. 
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Figure 4.1 IFFN Application of MCES Module 1. 

The IFFN Testbed had focused its analysis on specific concerns of .Army, .Air 

Force, NATO, and DOD decision makers regarding the role of identification as it 

contributes to the effectiveness of the air defense process. The major concern 

expressed by the IFFN Joint Testbed was the determination of how the programmed 

system and weapon systems would operate together. The IFFN mission and its 

environment (friends, foe, neutral, weather) had already been specified prior to Major 

Gandee's MCES application. The friendly weapon systems were limited to S.AMs and 

fighters with beyond visual range (BVR) munitions. A conventional threat scenario 

-> ' 

was already chosen by the IFFN Test Force so that stress on the system could be 
affected by vaiying traffic volume, ECM jamming to radars, communication jamming 
and varv’ing weather conditions. [Ref. 1: p. 65] 

The final step addressed by Major Gandee in this module was the analysis 
objective. The overall air defense C analysis objective was reformulated by the 
January 1985 MORS workshop from IFFN issues. The analysis objective [Ref 2; p. 1] 
was to determine: 

'y 

How effective is the air defense C'^ svstem in the central region in Europe in 
providing decisionmakers the means to assess and employ air defense assets to 
meet overall mission objectives? 
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Major Gandee realized that identification can be afiected by the presence of a physical 
entity or an asset like the airborne command post or by procedures such as those used 
for passing identification information. The IFFN analysis objective was eventually 
expanded by Major Gandee to determine: 

1. How effective is the process w'hen the C2 structure and its attendant 
changes in tactics and procedures is varied. 

2 

2. How effective is the C process when physical entities are added or lost. 
[Ref 2: p. 3] 

The details of formulating the analysis objective involved interaction between the 
decisionmakers, operational users and analysts. [Ref 12: pp. 21-23] 

C. C- SYSTEM BOUNDING 

In the next module, the bounding of the system of interest w'as confirmed by 
Major Gandee from previous IFFN efforts. Figure 4.2 lists the condensed results of 
implementing Module 2 for the IFFN Testbed. Physical entities were identified and 
bounded and Figure 4.2 depicts the successful application of the "onion skin" idea. 
Alternative organizational structures were determined and hierarchal charts formulated. 
.Again, much of this had been accomplished earlier at the IFFN Testbed but not by 
using specific MCES methodology. The application of this module confirmed that the 
IFFN C“ bounding was sound. [Ref 12: pp. 21-26] 

D. C- PROCESS DEFINITION 

1. Air Defense C“ Process Functions 

In this module. Major Gandee defined the process functions of the 

2 

distributed C2 air defense system. The air defense C process functions were 

determined to be: detect (D), track (T), identify (ID), assess threat (TA), assign 

weapon (WA), allocate weapon (AW), and weapons monitor and control (C). Figure 

2 2 

4.3 depicts the air defense C'^ process. These air defense C process functions were 
mapped to the modified Lawson's C process model to ensure that all C functions 

had been considered and Figure 4.4 depicts that translation. [Ref 12: pp. 32-37] 

2 

These air defense functions represented what the air defense C system and 

weapon system are required to accomplish together to perform the mission. For the 

2 

IFFN Testbed application, a process function was added to Law'son's generic C loop 
and the plan function was eliminated. Lawson's plan function did not correspond to a 
real-time activity during the IFFN execution phase, however, non-real time plans such 
as airspace control procedures (ACPs) and rules of engagement (ROE) are a part of 
weapons assignment, allocation, and control. 
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Figure 4.2 IFFN Application of MCES Module 2. 
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Figure 4.3 IFFN Air Defense C" Process Functions. 




Figure 4.4 



Mapping of Air Defense 



Functions. 
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2. Distributed C“ Process Interface 

Since the IFFN Testbed was dealing with a distributed C" system, the 
determination of process function boundaries was sometimes complex. Major 
Gandee discovered that in a distributed command and control system there are three 
distinct processes that will affect the overall performance of the system; intelligence, 
crosstell (coordination), and execution level processes. [Ref 12: pp. 38,39] 
a. XT ELL Process 

A separate Crosstell (XTELL) process provides a way to share target 
information for the purpose of improving the overall picture of the environment and 
improving the accuracy of information. This is especially important for identification 
(ID) information in the air defense application where command centers are 
geographically dispersed. Individual command centers may develop definitive ID 
information which can be used by other command centers who have a tactical 
advantage or resources to engage the target. The XTELL process is accomplished 
through three functions of Crosstell (XTELL), Track Correlation (TC), and ID 
Conflict Resolution (I DC). The XTELL process is represented in Figure 4.5 with its 
C“ process interface. 

The XTELL function of the XTELL process is the transfer and receipt of 
information via data link with some rules or filters. These rules specify where 
information is to be sent and what information will be received. The Track Correlation 
(TC) function resolves location and track numbering disagreements in the C^ system. 
The ID Conflict Resolution (IDC) function resolves conflicts that may arise in the 
identification process between different C^ nodes. At some nodes, this IDC function is 
a fusion process while at other nodes it is a decision process. 

Figure 4.6 represents the XTELL process in a lateral relationship. This 
lateral relationship represents adjoining units of the same level passing coordinating 
information between them. This information is then fused and correlated. A vertical 
or hierarchal XTELL relationship can also be present in distributed C^ systems. The 
vertical XTELL process is similar to the lateral relationship except that now the 
coordinating information flows between the hierarchical related units. The fusion and 
correlation of the identity and track information may be different than that in the 
lateral relationship since the higher level unit will usually have more voice in resolving 
conflicts of information. The alternative C^ systems have various configurations of 
these types of XTELL relations making some the configurations quite complex. 
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Figure 4.5 XTELL Process Functions and C" Process Interface. 
b. I .XTELL Process 

.An Intelligence (INTELL) process aids decisionmakers throughout the C“ 
system in forming perceptions of enemy capabilities and intentions. The INTELL 
process is accomplished through four functions of Sense (S), Process (P). Intelligence 
Correlation (IC), and Assess (A). [Ref. 12: pp. 39-42J 

The function which collects data necessary to describe and forecast the 
environment is termed the Sense (S) function. The function that transforms data into 
information about the enemy forces' disposition and actions is termed the Process (P) 
function. The Intelligence Correlation (1C) function correlates intelligence information 
with track and ID information. The Assess (A) function is performed when 
information is examined and patterns uncovered that indicate the actions or intentions 
of the enemy. The Assess function is also performed when patterns are utilized to 
forecast possible future changes in environment. Figure 4.7 graphically depicts the 
INTELL and C^ process along with the XTELL process. 
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Figure 4.6 Lateral XTELL Process. 



c. Execution Level C' Process 

The INTELL and XTELL processes support the Command and Control 
Process. The C“ process can be viewed at a level which directly controls weapon 
systems and at a higher echelon which coordinates the efTorts of C" processes which 
direct the weapon systems. Since the IFFN Testbed simulates the NATO air defense 
system which is geographically distributed, the C^ process included a netting of the 
separate command centers through the XTELL process. The INTELL process will 
also be interfaced with some of the process functions. The interfacing of the XTELL, 
INTELL, and C processes together by communication links, protocols, operational 
procedures determined the overall C architecture. Figure 4.8 lists the major actions 
completed in Module 3. [Ref. 12: pp. 44-46] 

E. INTEGRATION OF SYSTEM ELEMENTS AND FUNCTIONS 

Prior to developing measures, Gandee felt that a model or architecture that 
described the system was definitely needed. When Gandee attempted to establish an 
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Figure 4.7 INTELL. XTELL. and C“ Process Interface. 
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Figure 4.8 IFFN .Application of MCES Module 3. 
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architecture for the air defense C' system, he found that he needed a number of 
actions not listed in the then current MCES methodology. The methodology for 
interfacing all the processes into an architecture was not covered completely in the 
original version of MCES. The developers of MCES adopted this idea of the 
integration of system elements and functions into an architecture to support the C^ 
evaluation. 

1. Structures 

Information flow through the air defense C^ process functions was used by 
Gandee to derive a natural hierarchical relationship between the individual C" 
functions in the form of an information flowchart. Later command organizations and 
equipment and communications alignment were also related to form a organizational 
structure chart. [Ref. 9: p. 54] 

Major Gandee needed a methodology that documented this internal 
processing and described how the information is input to and output from the function. 
There are a number of methods available to formulate and describe the internal 
processing of C^ functions. In this module, a specific software design technique. Data 
Flow-Oriented Design [Ref. 7; pp. 99-115] was used to integrate the system elements 
and functions of the system. Thus, this input, output relationship could form a 
description of the internal information flow between separate process functions as 
required to perform the mission. The end result was a “structure" for a particular 
version of the system. The MCES definition of “structure" states that structure 
identifies the arrangement and interrelationships of physical entities, procedures, 
protocols, concepts of operation, and information patterns. [Ref 12: p. 48] 

a. Data Flow-Oriented Design 

In the first step, each process function was examined and the data 
flowing through the function defined. A graphic representation of this process is 
termed a data flow diagram (DFD) and they describe the input 'output relationships 
that exist between the C functions. Figure 4.9 depicts a data flow diagram (DFD) for 
an "execution level" C process at a single command node. The DFD's were also 
applied to interface the INTELL, XTELL, and Force processes with the process in 

the distributed IFFN C system. Major Gandee described how that information flow 
linked those separate processes into an architecture of the complete or combat 
system. [Ref 12: pp. 47,48] 
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Figure 4.9 Air Defense Single Node Data Flow Diagram. 
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b. Tramform Analysis 

In the next step, the C“ process as a whole was reviewed and a transform 
analysis performed on the DFD to determine the process or transform center. 
Using this flow of information into and out of each function, a transform analysis was 
conducted to determine the information transforming process center where information 
flowed in and were the information was transformed into an output in the form of 
control information. A function is analogous to a data flow transform. .An 
example is shown in Figure 4.9 which depicts the Assess Threat function as the 
process or transform center of the basic air defense process where the main perception 
is formed. Information flows in to formulate this perception and is termed the afferent 
branch. Information flows out in terms of decisions based on this perception and 
constitutes control flow and is termed the efferent branch. Structurally, this Assess 
Threat function subordinates the others and it was designated as the C process center. 
A structure chart was derived from this transform analysis which shows the overall 
structural relationships between each process function. This process center was 
then used to establish the structural hierarchy betw'een C process functions. This 
hierarchical relationship between C2 process functions w'as presented as a structure 
chart. .All these functions can be potentially performed in a single node or be 
distributed between different nodes. 

After the functional structure is formed, the people and their equipment 
can then be matched to that structure. Major Gandee's gave an example of the battle 
commander performing the Assess Threat function supported by the identification 
officer. The commander also has subordinate weapon assignment officers who 
implement his decisions to attack the most important targets. Major Gandee found 
that the equipment consoles can be matched to this structure as capabilities to assign 
targets or control weapons can be implemented by configuring consoles to address 
their output in accordance with the specified "structure". [Ref. 12: pp. 48-50] 

c. Null Process Concept 

Under some operational concepts, process functions can be distributed 
between command nodes such as Brigade and Battalion FDCs or between command 
nodes and weapon systems such as the CRC and fighter. The C2 process functions can 
be divided. Such arrangements are often temporary and unique to the particular 
version of the system. Major Gandee developed the concept of a null process to 
differentiate between the C process functions when they are distributed. For example. 
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the Brigade FDC allocates to the Battalion FDC and the Battalion FDC allocates the 
weapon system. Only one C" process can direct a weapon system although its 
decisions may be influenced by information coming from other processes. Influence 
can come in the form of an indirect ID or priorities from a higher echelon. Each 
command node can potentially perform all functions to direct force actions in the 
environment. Figure 4.10 and Figure 4.11 depict the distribution of and force 
functions between a Battalion FDC and SAM battery for two differing operational 
concepts of centralized and decentralized control. A function can be null at a facility 
due to a physical limitation such as the null Detect function at the Battalion for lack of 
organic radar or due to a redistribution of decision functions to reflect a different 
operational doctrine. 

With these techniques, the C system architecture was changed to show 
relationships between "physical entities", and "processes", to produce a "structure". 
This structure was altered to reflect different operational concepts which form the 
different versions of the system. There should be a different structure for each 
alternative system. Figure 4.10 is a good example of the structure of a battalion 
employing centralized control of its battery fire units and is different than the structure 
in Figure 4.11 of a Battalion employing decentralized control of its battery' fire units. 
In these illustrations, the null functions are not enclosed by a box. (Ref 12: pp. 51-53) 
d. Procedures 

Procedures are utilized in the internal processing within C functions. For 
instance, some IFFN issues deal with ID value of air space control procedures. These 
rules or procedures are specified externally but used internally within the ID function 
to determine ID. These rules, when combined with other sources for ID into some 
decision loop or algorithm, affect the internal "structure" of the ID function. If these 
procedures are taken away then the decision loop (internal structure) is changed. 
Major Gandee used a design technique which provides a module description that 
explodes each function to define the internal processing and the coupling to other 
or force functions. In this approach the functions were related to an appropriate 
physical entity prior to determining relevant measures of performance (MOP), 
measures of effectiveness (MOE), and measures of force effectiveness (MOFE). 
[Ref 9: pp. 54,55] 
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Figure 4.10 Centralized Control of a Battalion Fire Unit. 
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Figure 4.1 1 Decentralized Control of a Battalion Fire Unit. 
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2. Architecture 

The MCES defines "architecture" as an description of a integrated set of 
systems whose physical entities, structure and functions are coherently related. The 
architecture provides a representation which will eventually lead to the ability to 
measure the system response and the effectiveness of directing forces to accomplish 
the mission. The integration of the system elements of man and machine with the 
process functions will eventually form an overall architecture that can be used by 
analysts to evaluate the system. The final step is to formulate a overall system 
architecture that will incorporate all the different version structures internally. The last 
step in completing the architecture was to identify what physical entites performed the 
individual process functions and what connectivity linked the functions together. This 
established a single architecture represented as an overall structure chart. The final 
form of the architecture includes the process description and system elements 
performing the processes arranged in a structural framework. Additional modules to 
the structure chart provided documentation for equipment, personnel requirements, 
and connectivity in the necessary’ detail. 

The general architecture will remain unchanged but the structure variations 
would be represented by the different version's unique information flow. An air 
defense example was illustrated by Major Gandee to describe how structures can differ 
internally in a system architecture. This illustration involved air defense operators 
located in front of consoles. Equipment consoles could be configured in various ways 
to aid the operator in performing certain functions and allow the output to be 
transfered to other consoles. The operator would be aided in his ability to process 
information and communicate it through a machine structure that parallels an 
organizational structure. The general C architecture would be the same but there 
would be unique structures utilizing the equipment and personnel differently. 
[Ref 12; pp. 48-50] 

Major Gandee listed three advantages for utilizing an architectural 
representation of the system. Major Gandee found that the process can be 
broken down into separate functions and appropriate attributes defined more 
systematically than previous brute force or exhaustive listing methods. For example, 
the major Identify function attributes were relatively easy to determine as accuracy, 
timeliness and completeness. The second advantage of an architectural representation 
is the cabability of defining where a measure should be taken to measure a certain 
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function. Certain operational concepts have the same function performed at dilferent 
nodes or levels. Therefore, measurements must take place where the function is being 
conducted. For example, the Allocate Weapons (AW) function is performed at the 
Battalion level in the Centralized Control mode, Figure 4.10 , but is performed at the 
Fire Unit level, Figure 4.11 , in the Decentralized Control mode. The Battalion just 
monitors the .Allocate Weapons function activities in the Decentralized Control mode. 
If an accurate architecture depicts the actual operations of the C" system, these 
relationships are clearly delineated. .A third ver>* important reason for architectural 
representation is its capability to graphically depict the C" system and weapon systems 
and highlight appropriate operational issues. [Ref. 9: pp. 54-56] 

Major Gandee's work did result in the addition of more objectives and 
ultimately additional measures to Test Series 2 as new relationships were uncovered. 
Figure 4.12 displays the major results for the implementation of Module 4 to the IFFN 
Testbed. 
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Figure 4. 12 IFFN Application of MCES Module 4. 



F. SPECIFICATIONS OF MEASURES 

Major Gandee used a different approach than the IFFN Testbed when 
determining the measures needed to evaluate the competing versions of the air defense 
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C“ systems. Instead of moving from issues to objectives to measures to answer those 
objectives, Major Gandee examined the information How and bounded system elements 
to determine which measures were needed. Figure 4.13 graphically depicts .Major 
Gandee's approach for determining measures for the air defense system. In this 
figure, the shaded area represents the functional air defense system. The MOPs are 
measures of each function within the system. Each function is dependent on those 
functions preceding it, so the .MOPs are conditional probability measures with 
additional range and timeliness measures not shown. MOEs measure how well the C“ 
system performed all its functions as a whole and is measured outside of the system. 

1. MCES Test Series 2 Issues 

Major Gandee illustrated some of the issues he considered concerning 
identification, centralized control, and network, connectivity for Test Series 2. Focusing 
upon such issues lead to the differentiation among the alternative architectures. 

• Issue 1: Will centralized control at the Battalion manage the missile resources 
better by spreading the fire power more evenly over subordinate units and over 
time? 

• Issue 2: Under what traffic volume conditions can centralized control be 

handled without degradation? 

• Issue 3: If the data links (XTELL process) earn-' information on which targets 
have been allocated for engagement, can SAM batteries operate in a self- 
deconflicting manner conserving missile resources? 

• Issue 4: Given decentralized control and self-deconflicting doctrine, is a fullv 
connected data network required to prevent a single point failure due to the 
possible destruction of a Battalion Firing Unit? 

• Issue 5: Will the XTELU network supply the most complete ID to the other 
S.-\M batteries when their ID equipment becomes inoperable? (Ref 9; pp. 
57.58] 

2. MCES Measures 

Major Gandee's first four issues for Test Series 2 were sensitive to structural 
changes. Major Gandee suggested possible efficiency and coordination measures which 
would reflect the probability of a target being engaged by more than one unit due to 
lack of coordination. The fifth issue of identification questioned whether a network 
could increase individual unit identification capabilities. This could be accomplished by 
supplying more complete ID information from other units which formulate the ID 
information. Major Gandee suggested that an ID accuracy measure could be used to 
compare the accuracy of an ID formulated at an organic unit and the ID information 
which passes over the netw'ork to other units as a system ID. Examples of generic 
MOEs are; timeliness, accuracy, survivability, capacity, and percent completion. 
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SUCCESSFUL 



Figure 4.13 MCES Approach for Determining .Measures. 
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Other possible measures suggested by Major Gandec were fusing measures that 
measured the ability of units to accept and fuse system ID information with its own 
organic ID information to improve the ID accuracy in time to use it elTcctively. Figure 
4.14 represents the major measures recommended by Major Gandee during the MCES 
application of Module 5. [Ref. 12: pp. 71-76] 




Figure 4.14 IFFN .Application of .MCES .Module 5. 



G. MCES APPLICATION SUMMARY 

Major Gandee's proposed application of .MCES to the IFFN basically stopped at 
Module 5, Specification of Measures, due to time constraints and the delay of Test 
Series 2 execution. As a result, the MCES data generation and aggregation and 
interpretation modules were not evaluated for Test Series 2 by Major Gandee. The 
MCES methodology provided a evaluation methodology to assist the IFFN Test Force 
in its evaluation of the air defense C^ problem. The MCES approach of systematically 
outlining the physical entities, structure, and C^ process functions insured that all 
evaluation areas w’ere covered. MCES has definitely helped in highlighting the 
functional measures that have beea overlooked in previous C^ evaluations. The 
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distributed functions are required to characterize the coordination and intelligence 
sharing of distributed systems. There are difTerent levels of these distributed 
functions and the structures can become ver>' complex. MCES's distributed 
functions did assist in describing these complexities. The MCES concept of using a 
model or architecture to establish a baseline with alternative structures to represent the 
competing C“ systems was very useful in developing measures to differentiate between 
them. Understanding the system has to take place before attempting to measure its 
utility and to uncover which variables are responsible. The MCES approach appeared 
to be detailed and complete and new relationships were uncovered by Major Gandee. 
MCES provided the IFFN C^ analysts with a theoretical framework for determining 
the utility of a C^ system. The MCES application did assist the IFFN Testbed. 
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V. ANALYSIS OF THE IFFN TESTBED AND MCES APPROACHES 



A. FEEDBACK TO THE DECISION-MAKER 

Major Grey, Chief, Operational Analysis section, IFFN Joint Test Force, and the 
IFFN testbed director, Colonel Dave Archino, participated in the MORS workshop 
and later incorporated some of the MCES ideas into its test plans. Major Gandee 
visited the IFFN Testbed, conducted his research, and made his MCES application to 
the IFFN Testbed with the full assistance of Major Grey. Major Gandee's proposed 
application of MCES to the IFFN basically stopped at Module 5, Specification of 
Measures, due to time constraints and the delay of Test Series 2 execution. As a 
result, the MCES data generation and aggregation and interpretation modules were not 
evaluated for Test Series 2 by Major Gandee. Due to even another delay in the 
execution of Test Series 2 until March 1987, the final results were not evaluated during 
the conduct of this thesis as was originally planned. 

Test Series 1 was planned and conducted without the aid of MCES. The early 
planning stages of Test Series 2 had been completed before the MCES application 
started. However, when new or better ideas were developed in the MCES application, 
some of these ideas were added to the Test Series 2 test plan. A strict comparison of 
the IFFN Testbed produced Test Series I results to Test Series 2 would not be valid 
due to the mi.xed participation in Test Series 2. In the evaluation of Test Series 2. it 
was sometimes difTicult to determine exactly what part of the test plan was attributable 
to MCES or the IFFN Testbed. In almost all cases, MCES at least confirmed earlier 
IFFN Testbed planning. In some cases, MCES provided new insights that were 
responsible for a better test plan. 

B. PROBLEM FORMULATION 

.Mthough the problem formulation was completed earlier by the IFFN Test 
Force, MCES was used to verify that the correct steps were taken. As previously 
noted, the analysis objective was expanded by the MCES application. The problem 
formulation revolved around the air defense problem and not around how to build a 
credible testbed to evaluate competing air defense C^ systems. The IFFN Testbed 
itself was a system that could be evaluated just as the IFFN Testbed was trying to 
evaluate air defense C^ systems. The IFFN Testbed was evaluated as part of its 
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certification effort. Measures were formulated that would eventually determine if the 
IFFN Testbed produced valid and credible results. .A comparison of the results was 
conducted between the values of the measures of target identification, allocation, and 
engagement from the IFFN Testbed simulation and actual Patriot livefire exercises. 
The building of the testbed itself was a background problem and will be covered in 
greater detail in the data generation discussion. 

C. C- SYSTEM BOUNDING 

The bounding of the system of interest was confirmed by Major Gandee from 
previous IFFN efforts. Physical entities were already identified and bounded. Much of 
this had been accomplished earlier at the IFFN Testbed but not by using the specific 
MCES methodology. The application of this module confirmed that the IFFN 
bounding was sound. The "onion skin" idea was a useful tool and was subsequently 
used by the IFFN Testbed to graphically display their bounding. 

D. C“ PROCESS DEFINITION 

The IFFN Testbed originally identified three functions that were performed in 
the air defense process and those were: identification, target allocation, and target 
acquisition. The IFFN Testbed did conduct a functional analysis of the air defense 
process though not in as much detail as the MCES functional analysis. Major Gandee 
and the MORS team defined the process functions of the distributed air defense 
system as: detect (D), track (T), identify (ID), assess threat (TA), assign weapon 
(WA), allocate weapon (.AW), control (C) which were later adopted by the IFFN 
Testbed. 

E. SYSTEM ELEMENTS AND FUNCTIONS 

The IFFN Testbed did formulate the alternative C systems that it wanted to 
test. Organizational and equipment charts were constructed as well as some 
information flow charts by the IFFN Test Force. The IFFN Testbed baseline criteria 
was basically a baseline architecture from which planned derivations would be tested. 
In this manner the IFFN Testbed accomplished an integration of system elements and 
functions but in less detail than Major Gandee's application. The .MCES application 
by Major Gandee was more thorough with his information flow charts, organizational 
charts, and structure charts. Alternative organizational structures were determined and 
hierarchal charts formulated by both the IFFN Testbed and MCES approaches. 
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however. Major Gandee's structures were more detailed and complete. Major Gandee, 
through his use of null functions and adding or deleting physical entities, developed 
many different configurations for possible testing. The alternative configurations for 
centralized and autonomous (decentralized) control of the Patriot Fire Units was one 
of Major Gandee's contributions. This concept was added to the IFFN Testbed's 
analysis. 

F. SPECIFICATION OF MEASURES 

General categories of measures were sought by the IFFN Testbed to derive 
values that would eventually lead to discriminating among the competing air defense 
C^ systems. Early in the development of the IFFN Testbed, the analysts realized that 
they could determine differences between the competing C^ systems without precisely 
knowing which variables were responsible for the difTerence. As the development of 
IFFN Testbed progressed, the analysts knew they needed to determine which variables 
were causing the difference. 

1. Deriving Measures 

When and where does the analyst derive measures in a C system? There 
were two different approaches considered by the IFFN Testbed as they derived 
measures to evaluate competing C^ systems. The MCES approach utilized the C^ 
architecture and unique structures to derive their measures. The MCES methodology 
built a baseline C^ architecture with alternative structures to derive where and w'hen 
the measures should be determined. The IFFN Testbed approach utilized issues, 
objectives, and subobjectives to derive their measures and later built a baseline 
architecture to determine where to use the measures. The Institute for Defense 
.Analysis (IDA) conducted extensive research in their attempt to determine measures 
for use by the IFFN Testbed [Ref 13]. IDA also basically used a functional 
decomposition of the air defense C^ system to derive its measures. Alphatec, Inc. 
developed a petri net model of the IFFN air defense C^ system and then identified 
measures to determine the characteristics of each interconnection in the net [Ref 14]. 
This massive effort resulted in over 200 measures for their five levels of the system. 

There were originally six objectives formulated for Test Series 2 that led to the 
IFFN Testbed measures. A new list of objectives was formulated after the MCES 
application and appeared in the next revised version of the Test Series 2 test plan and 
is listed below. Objectives 2. 6, and 7 were added to Test Series 2 after the MCES 
application uncovered the need for them. 
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• Objective 1 - Assess and contrast the performance of the PATRIOT battalion 
under centralized and decentralized control. 

• Objective 2 - Evaluate the contribution of adding (Patriot Battalion FDC ) to 
autonomous Patriot fire unit performance. 

• Objective 3 - .Assess the impact of changing and removing Airspace Control 
Procedures (.ACPs) on the operational perTormance of the centralized and 
decentralized battalion and autonomous fire unit. 

• Objective 4 - Detennine the value and impact of perfect ID and communication 
system performance on PATRIOT battalion performance. 

• Objective 5 - Evaluate the impact of various changes in direct and indirect ID 
periormance and interaction. 

• Objective 6 - Evaluate the contribution of Question and .Answer (Q&A) IFF 
devices to Patriot Bn and FU performance. 

• Objective 7 - Evaluate the abilitv of indirect ID information to compensate for 
the loss of the direct Q&.A IFF device at a Patriot FU. 

• Objective S - .Assess the influence of the fire unit ID function on the ablitv of 
the PATRIOT battalion to perform its functions. 

• Objective 9 - Identifv and subjectivelv evaluate anv P.ATRIOT operational 
deficiencies noted during testing. [Ref II: pp. 2. 1,2. 2]' 

2. MOEs versus MOPs 

The terminology used by the IFFX Testbed for MOEs and MOPs is directly 
opposite of the MCES terminology. MCES states that MOEs are measured outside of 
the C“ system and that MOPs are measures of the functions within the C“ system. 
MCES MOPs are used for the C system measurements. Examples of MCES .MOPs 
would be probability of detection and correct identification. Within the force 
boundary, MCES Measures of Effectiveness (MOEs) are used for measuring the 
functions of the C" system. Examples of generic MCES MOEs are; timeliness, 
accuracy, survivability, capacity, and percent completion. MOFEs are used for the 
boundary measurements between the force and the environment. .An MCES example 
might be the number of enemy aircraft destroyed prior to releasing their weapons. 

The IFFN Testbed termed the functional measures as MOEs and their 
corresponding submeasures as MOPs. The IFFX Testbed MOEs for Test Series 2 
would measure the needed information for the C^ evaluation objectives. .MCES termed 
these type of measures as Measures of Performance (MOP) since they were measures 
of the air defense functions. There is obvious disagreement in methodologies as to 
what to name the different set of measures. This was not an important detail and did 
not cause too much difficulty. 
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3. Functional Measures 

Major Gandee's approach was to utilize the functions as the focal point 
for deriving measures. If this approach is to be strictly followed, then each function 
would have at least one corresponding probability measure plus time and distance 
distribution measures. The measures of probabilities and time and distance ranges for 
each function should be the minimum measures used to measure these functions. 
Major Gandee proposed these measures and highlighted that all the functions should 
have corresponding measures. Table 4 lists the measures that should be included for 
the functions. 







TABLE 4 


MCES FUNCTIONS AND MEASURES 


FUNCTION 






DETECT (D) 




P(Detect) , 

Time and Distance 


TRACK (T) 




P( Track) , 

Time, and Distance 


IDENTIFY ( ID) 




P( Identify) , 

Time and Distance 


ASSESS THREAT 


(TA) 


P( Assess Threat), 
Time and Distance 


ASSIGN WEAPON 


(WA) 


P( Assign Weapon), 
Time and Distance 


ALLOCATE WEAPON (AW) 


P( Allocate Weapon), 
Time and Distance 


CONTROL (C) 




P( Control ) < 

Time and Distance 



Most of these measures are listed in the test design plan for Test Series 2. Major Grey 
of the IFFN Testbed does state that the data for all these measures will be available 
but some were not considered for Test Series 2. Some of these functions will be 
measured indirectly and the functional measures may be incorporated in later tests if 
the results of Test Series 2 reveals that they are needed. 
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4. Distributed Functions and Measures 

Major Gandee thought that the measures for the XTELL and INTELL 
processes must be used to effectively evaluate distributed systems. If this is indeed 
the case then each function should have at least one corresponding measure of 
performance. Examples of these are shown in Table 5 . .Most of these measures were 
used by the I FEN Testbed. The IFFN Testbed will use their measures of P(Pass), 
P(Res), P(Trans), and P(.Amp) to measure the indirect ID information flow which will 
indirectly measure some of the coordination and intelligence functions. These IFFN 
measures and definitions are listed for review and comparison. 

• P(Pass) Probability that a passed ID is correct. 

• P(Res) Probability that an ID conflict is resolved while the aircraft is still in 
the weapon system's measurement volume. 

• TjTrans) Distribution of times elapsed between receipt and retransmission of 
ID information by a C2 node. 

• PI-Amp) Probability that an ID includes track amplification information. 





TABLE 5 




DISTRIBUTED FUNCTIONS AND MOPS 




FUNCTION 


MEASURES 




CROSSTELL (XTELL) 


P( Correct Fusion) 




Track Correlation (TC) 


Timeliness, accuracy 




ID Conflict 
Resolution ( IDR) 


Timeliness, accuracy 




INTELLIGENCE ( INTELL) 


PfTarget Engagement, 
I Id was used) 
PfTarget Engagement, 
I Id was not used) 


given 

given 


EXECUTION LEVEL 
C2 PROCESS 


P( Correct ID prior to 
engagement) 
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5. Operational versus Design/Qualitative Measures 

While most evaluations of systems center around design qualitative 
measures such as flexibility, surviveability, availability, etc., the IFFN Testbed used an 
approach much like the MCES methodology in that the functions of the system were 
first studied before equipment and personnel problems were considered. These 
measures are sometimes refered to as operational measures. Examples of operational 
measures for distributed systems are: 

• CAPACITY 

• ACCLTCACY 

• RESPONSE TIME 

• AVAILABILITY 

• THRU PUT 

The IFFN Testbed did not continue their evaluation to include the design'qualitative 
measures that are definitely needed to determine the utility of a particular competing 
C“ system. Major Grey did state that design type measures would be incorporated in 
later tests and that the data needed for these types of measures was readily available 
even for Test Series 2 if the need arose or if a higher authority requested that 
information. Examples of design and quality measures are: 

• EFFICIENCY 

• RELIABILITY 

• SURVIVABILITY 

• USEABILITY 

• CORRECTNESS 

• .MAINTAINABLITY 

• VERIFIBILITY 

• EXPANDABILITY 

• FLEXIBILITY 

• INTEROPERABILITY 

• PORTABILITY 

• REUSABILITY 

• ROBUSTNESS 

• EVOLVABILITY 

Another analyst, Leslie Golliday, listed a set of generic air defense measures, 
Table 6 [Ref 15: p. 788], which are similar to what the MCES application had 
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determined. These measures are more general but the specific measures could be 
derived from the general measures. .Again, the list included function measures as well 
as design qualitative measures. 



TABLE 6 

C2 MEASURES FOR AIR DEFENSE 



MOP 


DEFINITION 


Alerting 

Capability 


Measures capability of providing 
gross positional data on an aircraft 
at extended ranges. 


Cueing 

Capability 


Measures the process of providing 
specific and timely positional 
data with tentative identification 
of an aircraft within a designated 
range of a unit. 


Weapons Control 

Information 

Capability 


Measures capability to provide 
weapons control order ( WCOs ) 
and rules of engagement (ROEs). 


Airspace 

Management 

Capability 


Measures capability to provide 
avoidance of engagement of 
friendly fixed and rotary 
aircraft. 


MOEs 





INTEROPERABILITY 

RELIABILITY 

MAINTAINABILITY 

FLEXIBILITY 

USE ABILITY 

SURVIVABILITY 



6. Resource Conservation and Reallocation Measures 

Major Gandee suggested possible efficiency and coordination measures which 
would reflect the probability of a target being engaged by more than one unit due to 
lack of coordination. A network may increase an individual unit's identification 
capabilities by supplying more complete ID information from other units which 
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formulate the ID information. .Major Gandee suggested that an ID accuracy measure 
could be used to compare the accuracy of an ID formulated at an organic unit and the 
ID information which passes over the network to other units as a system ID. Other 
possible measures suggested by Major Gandee were fusing measures that measured the 
ability of units to accept and fuse system ID information with its own organic ID 
information to improve the ID accuracy in time to use it eflectively. However, the 
suggested measures of weapon allocation elTiciency, unit ID coordination. ID accuracy, 
and ID fusing ability were not used by the IFFN Testbed. Some of these attributes 
could be measured indirectly using some of the other IFFN Testbed measures such as 
timeliness and the probability of correctly identifying an object to measure ID 
information fusing ability. 

Determining which competing consumes less missile resources is very 
important considering the cost and shortage of modern missiles. Also, in a dynamic 
environment, there will be instances when missile resources will have to be reallocated 
due to prior destruction of the target, previous allocation to another weapon, higher 
priority targets in the area or the targets flies out of range. .Measures of efficiency and 
reallocation ability seemed to be crucial differences in the competing C" systems. 
•Again, Major Gandee suggested such measures but they were not all incorporated into 
the current Test Series 2 plan. Research by .Alpha tec, Inc. with Petri nets also 
suggested reallocation measures for the IFFN Testbed [Ref. 14]. 

G. DATA GENERATION AND TESTBED DESIGN 

Major Gandee's proposed application of MCES to the IFFN basically stopped at 
Module 5, Specification of .Measures due to time constraints and the delay of Test 
Series 2 execution. As a result, the MCES data generation and aggregation and 
interpretation modules were not evaluated for Test Series 2 by Major Gandee. 

The current MCES does incorporate a experimental design methodology for 
building testbeds to generate data. A methodology. Systems Effectiveness Analysis 
(SE.A), was introduced by A. H. Levis and P. Derskin [Ref. 16] for evaluating large 
scale systems such as testbeds and ultimately integrated into the .MCES methodology. 
To produce valid data, a testbed must be credible and SEA was developed to assist in 
validating testbeds. SEA was also developed to determine the minimum number of 
experiments needed to evaluate the system and to formulate a optimal sequence of 
improvements areas for the competing configurations. Once the testbed was 
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determined to be credible, experiments could be run that would determine the optimal 
elTectiveness of each alternative system. 

One of Dr. Levis' students, Phillipe .Martain, demonstrated how SEA could be 
applied to the IFFN Testbed. First in the SEA process was the determination of the 
smallest number of experiments that were needed to be run to evaluate the 
elTectiveness of the testbed system. A simplified mathematical model utilizing 
Lanchester type combat models was developed to determine the minimum number of 
experiments required to evaluate the testbed. The testbed experimental results and the 
mathematical model results could also be compared to insure similar results. In a 
second phase, a system planning procedure was used to select the best evolution path 
for the testbed configurations from a fixed set of improvements. SEA can be then used 
in the last MCES module of Data Aggregation and Interpretation to make adjustments 
to the testbed experiments. [Ref. 17] 

H. OVERLAPPING OF PROCEDURES AND TOOLS 

The IFFN Testbed has been working on its air air defense problem for a 
number of years and through its iterative process evolved to a solution that was close 
to the MCES solution. The IFFN Testbed started without the aid of MCES and some 
of the applied .MCES methodology overlapped with the previous methods used by the 
IFFN Joint Test Force. However, in most cases both methodologies resulted in the 
same general results. The IFFN Testbed did make changes after the MCES 
application, but it is not clear if these changes will have a major impact on Test Series 
2 since the test has not been completed. 

MCES does integrate and imbed current evaluation tools. MCES does not 
preclude these tools and in fact uses them to obtain a better solution to the problem. 
Both the IFFN and .MCES methods came to the same general conclusions, however 
the .MCES approach appeared to be more complete. The MCES methodology 
attempts to standardize the analysis by providing a structured template to assist the 
analysts in their evaluations. 

I. Problem Formulation and Bounding 

The problem formulation and bounding of the C2 system were almost 
identical. Most system analysis methodologies start out in this same manner. 
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2. Functional Analysis 

Other system analysis approaches use the basic input, process, and output 
approach to describing the C*" system. Functional and process analysis are being used 
often in the software engineering environment. These approaches are similar to C. 
West Churchman's system process of input, process, and output as explained by 
Schoderbek. et al. [Ref 18: pp. 8-29]. It is interesting to note that major methodology 
revisions occured when analysts attempted to automate systems because they needed to 
precisely describe and recreate the functioning of the manual system. 

3. Model or Architecture Building 

The system analysis approach of utilizing the functions of the system to build 
a systems model has been used in a number of previous evaluations. Other researchers 
have added methods of modeling that can be integrated into the MCES methodology. 
A specific example already referenced is Systems Effectiveness Analysis (SEA). Dr. 
Levis has conducted stimulating research in this bottom-up approach and has 
integrated it into the MCES methodology [Ref 19). The SEA methodology insures 
that the simulation can simulate the whole range or limits of the interactions between 
the variables instead of a smaller subset of the interaction range or limit. This can be 
accomplished by taking established measures and and determining their minimum and 
maximum ranges. The other quantitative methods of SEA can also be applied to the 
IFFX Testbed. 

Petri Nets have been used by researchers and analysts to mathematically 
model the different structures of information flow in the C^ architecture in the form of 
off and on states which can be used to describe a C^ architecture with unique 
structures. Alphatec, Inc. [Ref 14] conducted a study of the IFFN Testbed and 
constructed a number of different level petri nets to determine what measures were 
needed to measure what kind and how much information flowed between all the nodes. 
Basically, each connection between the nodes was an opportunity to measure 
information flow. 

4. IFFN and MCES Measure Specification 

The IFFN Test Force used its issues to formulate their original measures. 
However, there were no major deficiencies in the test design of Test Series 1 which was 
completed prior to the MCES application. There was prior research conducted by the 
Institute for Defense Analysis [Ref 13] and Alphatec [Ref 14] concerning the 
evaluation of the IFFN competing C^ systems and they confirmed the measures 
derived by the MCES application. 



77 



1. COMPARISON SUMMARY 

The IFFN Testbed has already taken advantage of the good ideas generated by 
the MCES application and the Test Series 2 plan has incorporated some of the MCES 
concepts. Although each methodology used different and similar tools to evaluate the 
competing systems, both approaches came to the same general conclusions. The 
question of whether the amount of time needed to document all possible interactions in 
the MCES is really needed is still unanswered. However after utilizing the VICES 
approach of analyzing the physical entites, organizational structure, and C functions 
in a systematic methodology, the IFFN Test Force discovered a number of important 
measures that they had not focussed on earlier in the test design process for Test Series 

2. Only with more testing and comparisons will the true value of the MCES approach 
to the IFFN Testbed be known. Conclusions and recommendations concerning both 
the IFFN Testbed and MCES are included in Chapter VI of this thesis. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 



A. IFFN CONCLUSIONS 

It is clearly evident that the IFFN Testbed has made some progress in solving 
some of the IFFN air defense problems. The problem is definitely complex and the 
IFFN Testbed has understandably committed large amount of resources to the 
problem. The Testbed is a good concept for an experimental design to test alternative 
C^ systems since all of the alternative systems can not be tested using actual 
equipment due to resource constraints or present C^ configuration limitations. The 
IFFN Test Force was careful to insure that the testbed was credible. Only one test 
series has been completed with Test Series 2 to begin in March 1987. The IFFN Joint 
Testbed has provided an excellent opportunity to test and refine MCES. The IFFN 
Testbed is still valuable as a suitable data generator for further evaluatation and 
refinement of MCES. 

The IFFN Testbed has already taken advantage of the good results generated by 
the MCES application and the Test Series 2 plan has incorporated some of the MCES 
ideas. After utilizing the MCES approach of analyzing the physical entites, 
organizational structure, and C^ functions in a systematic methodology, the IFFN Test 
Force discovered a number of important measures they had not focussed on earlier in 
the test design process for Test Series 2. The test design issues and measures were 
modified to accommodate the newly found distributed relationships between C^ nodes 
that were originally not formulated by the IFFN Joint Test Force. 

B. IFFN RECOMMENDATIONS 

An additional measure approach would be to utilize both operational and 
equipment design, quality measures. The functional measures are operational measures 
and the design, quality measures are more machine and resource oriented. The IFFN 
Testbed seems to have focused its measures on operational or functional measures and 
has not taken full advantage of available and possibly critical design qualitative 
measures such as resource efficiency. 

The IFFN Testbed should consider Vlajor Gandee's recommendations on the 
additional measures of performance and effectiveness, particularly the measures of 
coordination and resource allocation. Resource allocation, connectivity, availability. 
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surviveability. sustainability, and flexibility data appears to be available from the data 
collection points in the simulations. The measures for the XTELL and INTELL 
processes must be used to effectively evaluate distributed systems. 

C. MCES APPLICATION CONCLUSIONS 

The I FEN Testbed has been working on its air defense C^ problem for a number 
of years and through its iterative process evolved to a solution that was close to the 
same results. There is a learning curve associated with applying any new methodology 
which was quite evident in this I FEN application. During the MCES application to 
the I FEN Testbed, there was overlap of the evaluation tools used by the I FEN Test 
Force. Some of the tools used prior to the MCES application to the IFFN Testbed 
were included in the .MCES methodology. MCES does not preclude these tools and in 
fact uses them to aid in the evaluation to obtain the best results. This MCES 
application was a good start in the evolving of a generic standard C^ system evaluation 
method. The MCES approach of systematically outlining the physical entities, 
structure, and C" process functions insured that all areas were covered. Major 
Gandee's proposed application was used by the IFFN Operational Analysis section to 
better understand their air defense C^ problem. The MCES approach seemed to be 
more detailed and complete. New relationships were uncovered by Major Gandee 
resulting in the addition of new issues and measures. .MCES has definitely helped in 
highlighting the functional measures that have been overlooked in previous C“ 
evaluations. MCES did assist the IFFN Testbed. 

1. Integration of System Elements and Functions Module 

Major Gandee uncovered the need for the additional module 4. Integration of 
System Elements and Functions, that was not originally conceived as a part of the 
MCES modules. A model or architecture was needed to establish a baseline with 
alternative structures to represent the competing C^ systems. The system must be 
understood before attempting to measure its utility and uncover the variables are 
responsible for significant differences. By actually using MCES, some problems 
surfaced which ultimately resulted in refinements to the MCES methodology. 

2. Distributed C^ Process Functions 

The distributed functions are required to characterize the coordination and 
intelligence sharing of distributed C^ systems. There are different levels of these 
distributed functions and the C“ structures can become very complex. MCES's 
distributed functions assist in describing these complexities. 
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3. Solid Evaluation Tool 

MCES appears to be a viable evaluation template of current and evolving 
tools based on solid theorv’. MCES has provided the analyst community with a 
theoretical framework for determining the utility of a C" system. 

4. MCES Application to Additional IFFN Testbed Test Series 

The IFFN Testbed did revise their Test Series 2 plan to incorporate most of 
.Major Gandee's results of applying MCES. .Vlajor Grey has also utilized some of the 
concepts of MCES to formulate the design plan for Test Series 3. As more test series 
are completed, a more through evaluation of MCES can be made. 

D. MCES RECOMMENDATIONS 

1. Continued MCES Application to the IFFN Testbed 

The application of MCES to the IFFN Testbed should be continued on Test 
Series 2 and following tests. Due to time constraints and the delay of the Test Series 2 
execution, the final results were not available for analysis in this thesis as was originally 
envisioned. The areas of actual data generation and aggregation of measures and 
interpretation for Test Series 2 still could be a valuable thesis topic for a follow-on 
project. This follow-on project could document and evaluate the success of the 
previous implementation of .MCES and observe new implementations. A comparison 
of Test Series 1 with Test Series 2 results might reveal the differences in the approach 
and the results with the caveat that Test Series 2 was a slightly different simulation 
than Test Series 1. 

2. Further MCES Testing and Refinement 

More beginning to end C^ applications of MCES should be conducted to 
continue the evolution of the MCES methodology. 

3. Integration of More C“ Evaluation Tools 

.MCES does integrate a number of tools and could still integrate more while 
maintaining its solid theoretical base. The MCES approach is a top-down systems 
approach with certain advantages and disadvantages. Utilizing Dr. Levis' experimental 
design and bottom-up approach. Systems Effectiveness Analysis (SEA), would benefit 
the MCES toolbox. Petri nets show promise of being a good analyst tool for 
evaluating information flow. Other system evaluation methods should also researched 
for addition to the MCES methodology. 
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4. Standard MCES Terminology and Definitions 

MCES developers must decide on standard terminology and definitions to 
avoid confusion. Although MCES should be robust and flexible to analysts, a firm 
evaluation theory must be presented in simple, standard terminology. 

5. Education and Dissemination 

There is a learning curve associated with any new or revised methodology 
which was quite evident in this IFFX application. .■Xfter a standard terminology is 
defined, MCES should be announced and advertised to the C^ analyst and decision 
maker community because of its sound theoretical background. More disclosure of 
MCES is definitely needed. Since MCES does incorporate a number of known tools, 
MCES should be advertised as a structured approach to utilize C^ evaluation methods 
and tools. 
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